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The findings from this review do not reflect the full body of research evidence on Higher Achievement. 


What is this study about? 

The study examined the effects of Higher Achievement, 
a multi-year summer and after-school program for 
incoming fifth and sixth graders attending schools in 
at-risk communities. The program lasts through eighth 
grade and aims to improve academic achievement 
and encourage matriculation into an academically 
competitive high school. The study authors examined 
outcomes including: a) student achievement in reading 
and math up to 4 years after program participation; and 
b) high school choice activities— application to, admit- 
tance to, and matriculation at four different types of 
high schools with varying selection criteria. 2 ’ 3 

Three cohorts of incoming fifth and sixth grad- 
ers in Washington, DC and Alexandria, Virginia, 
were recruited for the study in 2006, 2007, and 
2008. About 950 students were randomly assigned 
through a lottery to an intervention group that was 
offered the Higher Achievement program or to a 
comparison group that was not. Just over half of the 
students were offered the opportunity to participate 
in the program. The analysis included over 800 stu- 
dents. To measure the program’s impacts, the study 
authors compared outcomes of the two groups. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

This study is a well-executed randomized controlled 
trial with low levels of sample attrition. 


Features of Higher Achievement 


Higher Achievement is a multi-year program 
operated out of centers that run from the summer 
before fifth or sixth grade through the end of eighth 
grade. The program includes a Summer Academy 
that offers core classes in math, science, social 
studies, and literature; two electives; weekly field 
trips; and academic competitions. In the summer 
before eighth grade, students also receive supports 
including group discussions, school visits, and 
mentoring sessions to prepare them for selecting 
and applying to high schools. The program also 
includes an Afterschool Academy that provides 
academic instruction by a volunteer mentor; an 
elective; help with homework; dinner; a community 
gathering; monthly field trips; community service 
projects; and, in the fall of eighth grade, preparation 
for selecting and applying to high schools. 

What did the study find? 

The study authors found that 4 years after randomiza- 
tion, students who were offered participation in Higher 
Achievement had significantly higher standardized test 
scores in mathematical problem solving. They were also 
significantly more likely than comparison students to be 
admitted to and matriculate at private high schools, and 
were less likely to apply to, be admitted to, and matricu- 
late at noncompetitive public charter/magnet schools. 
No statistically significant differences were found for 
standardized tests of reading comprehension; applica- 
tion to private schools; application to, admittance to, 
or matriculation at competitive public charter/magnet 
schools; or matriculation at neighborhood public 
schools. The magnitude and statistical significance 
of these findings were confirmed by the WWC. 
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Appendix A: Study details 

Herrera, C., Grossman, J. B., & Linden, L. L. (2013). Staying on track: Testing Higher Achievement’s 
long-term impact on academic outcomes and high school choice. New York: MDRC. 

Additional source: 

Linden, L. L., Herrera, C., & Grossman, J. B. (201 3). Achieving academic success after school: A 
randomized evaluation of the Higher Achievement program (Working Paper). Retrieved from 
http://www.leighlinden.com 


Setting The study was conducted in five Higher Achievement centers in Washington, DC and 
Alexandria, Virginia. 

Study sample Researchers recruited incoming fifth and sixth graders for the study through local media and 
referrals in the spring of 2006, 2007, and 2008, forming three cohorts. To be eligible, students 
were required to complete an application, attend an interview, be classified as “academically 
motivated” by Higher Achievement staff, and commit to participate for 3 to 4 years. Random 
assignment was stratified by the following factors: the Higher Achievement center to which 
the student applied, grade, gender, whether the student applied with a sibling, and baseline 
problem solving score. Siblings applying together (10.3% of students) were randomly assigned 
as a pair to the same experimental group; siblings of existing attendees were automatically 
admitted to the program and excluded from the sample. The amount of oversubscription to 
the program varied by year, so lottery assignment rates were adjusted for each cohort to fill 
the fixed number of available positions. In the first cohort, two-thirds of the students were 
assigned to the intervention group and one-third to the comparison group; in the second and 
third cohorts, half were assigned to the intervention group and half to the comparison group. 

Across the three cohorts, 952 students were randomly assigned, with slightly more than half 
beginning the program in the summer before fifth grade and the rest beginning in the summer 
before sixth grade. Seventy-five percent of students in the study were African American, and 
13% were Latino. Fifty-nine percent of students were female. About 60% of students were 
eligible for free or reduced-price lunch, 28% lived with both parents, and 12% spoke a language 
other than English at home. Sixty-five percent of students who applied to be in the study reported 
earning mostly A’s or A’s and B’s in school, while 6% reported earning mostly C’s or lower. The 
authors indicated that the students in the study scored marginally above the national average 
on the baseline standardized tests. 


Intervention The intervention group was offered an opportunity to participate in Higher Achievement: which 

group is a multi-year program with a primary focus on math and reading achievement, along with 

assistance preparing for high school application and selection. Starting in the summer after 
study enrollment (that is, before entering fifth or sixth grade) and continuing through the end 
of eighth grade, the intervention included a Summer Academy and an Afterschool Academy. 
The Summer Academy was offered 8 hours/day, 5 days/week, for 6 weeks. Classes of about 
13 students received core classes in math, science, social studies, and literature; two electives; 
weekly field trips; and participation in academic competitions. In the summer before eighth 
grade, students also received supports including group discussions, school visits, and men- 
toring sessions to prepare them for selecting and applying to high schools. The Afterschool 
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Intervention 

group 

(continued) 

Academy was held for 4.5 hours/day, 3 days/week, for 25 weeks, and included 75 minutes 
of academic instruction by a volunteer mentor, an elective, help with homework, dinner, a 
community gathering, monthly field trips, and community service projects. Each center served 
about 85 students. Of those offered the program, approximately 75%, 70%, and 47% of stu- 
dents attended the program after 1 year, 2 years, and 4 years, respectively. 

Comparison 

group 

Comparison students did not participate in Higher Achievement. The study authors did not 
provide further information. 

Outcomes and 
measurement 

Student achievement in math and reading was measured using the abbreviated versions of the 
Reading Comprehension and Math Problem Solving sections of the Tenth Edition of the Stanford 
Achievement Test (SAT-10). Tests were administered in the spring during the first, second, and 
fourth years after random assignment. Since the intervention was a multi-year program run- 
ning from the summer before fifth or sixth grade through the end of eighth grade, the outcomes 
measured 4 years after random assignment reflect the maximum exposure to the intervention 
and are used to determine the WWC effectiveness rating. 

High school choice outcomes were measured 4 years after random assignment through items 
on a parent survey that requested the names of high schools to which their children applied, 
were admitted, and matriculated. The researchers then coded each school as a private school, 
competitive public charter/magnet school, noncompetitive public charter/magnet school, 
or neighborhood public school. Competitive magnet schools were defined as those with an 
academic focus. Competitive charter schools were those classified as Tier 1 by the Washing- 
ton, DC public school system based on criteria such as high test scores and graduation rates. 
All other charter and magnet schools were classified as noncompetitive. Ten dichotomous 
variables were then created, indicating whether students applied to, were admitted to, and/ 
or matriculated at each of the first three types of schools, and whether they matriculated at 
a neighborhood public high school (which does not require application or admission). For a 
more detailed description of these outcome measures, see Appendix B. 

Support for 
implementation 

The intervention was delivered by staff in the Higher Achievement centers. In the Afterschool 
Academy, volunteer mentors taught in small group academic sessions. In the Summer Academy, 
paid teachers taught the courses. During eighth grade, the program’s manager of school placement 
visited each center biweekly to answer student questions and help with high school applications. 

Reason for 
review 

This study was reviewed by the WWC in response to a request by the Institute of Education 
Sciences (IES) and was suggested as a promising intervention through the WWC website’s 
help desk. 
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Appendix B: Outcome measures for each domain 


Mathematics achievement 

Abbreviated version of the Math Problem 
Solving section of the Tenth Edition of 
the Stanford Achievement Test (SAT-10) 

The SAT-10 is a group-administered, norm-referenced test. The abbreviated version of the Math Problem 
Solving section was used. Students’ scores on the tests taken 4 years after random assignment were used 
to determine the WWC effectiveness rating as they reflected maximum exposure to the intervention. 

Reading achievement 

Abbreviated version of the Reading 
Comprehension section of the SAT- 10 

The SAT-10 is a group-administered, norm-referenced test. The abbreviated version of the Reading Comprehension 
section was used. Students’ scores on the tests taken 4 years after random assignment were used to determine 
the WWC effectiveness rating as they reflected maximum exposure to the intervention. 

High school choice 

Type of high school to which a student 
applied 

Four years after random assignment, researchers surveyed parents of ninth graders, asking them to name the 
high schools to which their children applied. Each school was classified as either: (a) private, (b) competitive 
public charter/magnet, (c) noncompetitive public charter/magnet, or (d) neighborhood public school. Three 
binary variables were then created to indicate whether the student applied to a private school, a competitive 
public charter/magnet school, and/or a noncompetitive public charter/magnet school. (No variable was created 
for neighborhood public schools, because students do not apply to such schools.) 

Type of high school to which a student 
was admitted 

Four years after random assignment, researchers surveyed parents of ninth graders, asking them to name 
the high schools to which their children were admitted. Each school was classified as either: (a) private, 

(b) competitive public charter/magnet, (c) noncompetitive public charter/magnet, or (d) neighborhood public 
school. Three binary variables were then created to indicate whether the student was admitted to a private school, 
a competitive public charter/magnet school, and/or a noncompetitive public charter/magnet school. (No variable 
was created for neighborhood public schools, because admission is not required to attend such schools.) 

Type of high school at which a student 
matriculated 

Four years after random assignment, researchers surveyed parents of ninth graders, asking them to name 
the high school at which their children were matriculated. Each school was classified as either: (a) private, 

(b) competitive public charter/magnet, (c) noncompetitive public charter/magnet, or (d) neighborhood public 
school. Four binary variables were then created to indicate whether the student matriculated at a private school, 
a competitive public charter/magnet school, a noncompetitive public charter/magnet school, or a neighborhood 
public school. 


Table Notes: Twenty-one additional outcomes were examined in this study but were not included in this report because they were identified as context, fidelity, or attitudinal 
measures, and were therefore not considered of primary interest for this review. These include: (1) days per week spent in out-of-school-time (OST) programs in the summer, (2) 
days per week spent in OST programs during the school year, (3) community service or volunteer work, (4) speaking to a group outside of school about his or her ideas/work, (5) 
visiting a college campus to see what it would be like to be a college student, (6) reading books that are not for school, (7) writing not assigned at school, (8) visiting a business 
to see what it would be like to work there, (9) going to events outside of his or her neighborhood with his or her OST program, (10) participating in academic contests at his or her 
OST program, (11) attending a mock interview, (12) attending a test preparation class for the Secondary School Admission Test (SSAT) or High School Placement Test (HSPT), (13) 
practicing for the SSAT or HSPT but not as part of a class, (1 4) taking the SSAT or HSPT, (1 5) applying for a scholarship, (1 6) receiving a scholarship, (1 7) visiting a high school 
of interest, (18) speaking with teachers or other staff at a school of interest, (19) speaking with students who attended these schools about how they liked it there, (20) getting 
information about specific high schools, and (21) attending a shadow day at a high school. 
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Appendix C: Study findings for each domain 




Mean 

(standard deviation) 


WWC calculations 


Domain and Study 

outcome measure sample 

Sample Intervention 
size group 

Comparison Mean 

group difference 

Effect 

size 

Improvement 

index 

p-value 

Mathematics problem solving 

Abbreviated SAT-10 Math Year 4 

participants 

719 

students 

nr 

nr 

0.11 

0.11 

+4 

<0.05 

Domain average for mathematics problem solving 




0.11 

+4 

Statistically 

significant 

Reading comprehension 

Abbreviated SAT- 10 Reading Year 4 

participants 

719 

students 

nr 

nr 

0.04 

0.04 

+2 

>0.05 

Domain average for reading comprehension 




0.04 

+2 

Not 

statistically 

significant 

High school choice: Private school 

Application to private school Year 4 

participants 

719 

students 

27% 

21% 

6% 

0.20 

+8 

>0.05 

Admission to private school Year 4 

participants 

719 

students 

21% 

14% 

7% 

0.30 

+12 

<0.05 

Matriculation at private school Year 4 

participants 

719 

students 

15% 

9% 

6% 

0.35 

+14 

<0.05 

Domain average for high school choice: private school 




0.28 

+11 

Statistically 

significant 

High school choice: Competitive public charter/magnet school 

Application to competitive Year 4 

public charter/magnet school participants 

719 

students 

54% 

55% 

-1% 

-0.02 

-1 

>0.05 

Admission to competitive Year 4 

public charter/magnet school participants 

719 

students 

46% 

49% 

-3% 

-0.07 

-3 

>0.05 

Matriculation at competitive Year 4 

public charter/magnet school participants 

719 

students 

39% 

43% 

-4% 

-0.10 

-4 

>0.05 

Domain average for high school choice: competitive public charter/magnet school 


0.07 

-3 

Not 

statistically 

significant 

High school choice: Noncompetitive public charter/magnet school 

Application to noncompetitive Year 4 
public charter/magnet school participants 

719 

students 

14% 

22% 

-8% 

-0.33 

-13 

<0.01 

Admission to noncompetitive Year 4 
public charter/magnet school participants 

719 

students 

10% 

16% 

-6% 

-0.33 

-13 

<0.05 

Matriculation at noncompetitive Year 4 

public charter/magnet school participants 

719 

students 

6% 

13% 

-7% 

-0.51 

-20 

<0.01 

Domain average for high school choice: noncompetitive public charter/magnet school 


-0.39 

-15 

Statistically 

significant 
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High school choice: Neighborhood public high school 


Matriculation at neighborhood Year 4 719 40% 35% 

public high school participants students 

5% 

0.13 

+5 

>0.10 

Domain average for high school choice: neighborhood public high school 


0.13 

+5 

Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded 
to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the 
WWC. nr = not reported. 

Study Notes: Corrections for multiple comparisons were needed but did not affect whether any of the contrasts were found to be statistically significant. The p-values presented 
here were reported in the original study. The means and standard deviations for SAT-10 scale scores were not reported by the authors. However, the authors standardized student 
test scores using national norms and recalibrated them to have a mean of 0 and a standard deviation of 1 , which means their results are in effect size units. Mean difference and 
effect sizes were regression-adjusted for pretests, grade at baseline, gender, age, receipt of free or reduced-price lunch, race, and whether the student applied with a sibling; family- 
level controls including fixed effects for the center to which the student applied; and a set of baseline self-perceptions of abilities, peer academic support, and general adult support. 
The regression also includes cohort fixed effects (probabilities of random assignment differed by cohort) and standard errors clustered by family (the level of randomization). The 
study is characterized as having a statistically significant positive impact on the following domains: (1) mathematics problem solving, (2) high school choice: private school, and (3) 
high school choice: noncompetitive public charter/magnet school (Note - for this outcome, a negative impact estimate indicates a good outcome for students, because they are not 
attending non-competitive schools). For these three domains, at least one measure in each domain was positive and statistically significant and no effects were negative and statisti- 
cally significant, after accounting for multiple comparisons. The study is characterized as having an indeterminate effect on the following domains: (1) reading comprehension, (2) 
high school choice: competitive public charter/magnet school, and (3) high school choice: neighborhood public school. For these three domains, there were no statistically significant 
or substantively important findings reported. For more information, please refer to the WWC Standards and Procedures Handbook, version 3.0, pages 26-27. 
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Appendix D: Supplemental findings by domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Mathematics problem solving 

Abbreviated SAT-10 Math 

Year 1 
participants 

815 

students 

nr 

nr 

0.03 

0.03 

+1 

>0.05 

Abbreviated SAT-10 Math 

Year 2 
participants 

776 

students 

nr 

nr 

0.10 

0.10 

+4 

<0.05 

Reading comprehension 

Abbreviated SAT-10 Reading 

Year 1 
participants 

815 

students 

nr 

nr 

0.02 

0.02 

+1 

>0.05 

Abbreviated SAT- 1 0 Reading 

Year 2 
participants 

776 

students 

nr 

nr 

0.08 

0.08 

+3 

>0.05 


Table Notes: The supplemental findings presented in this table are additional findings that do not factor into the determination of the evidence rating. For mean difference, effect 
size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. The effect size is a 
standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students who are given the intervention (measured in 
standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percentile rank 
that can be expected if the student is given the intervention, nr = not reported. 

Study Notes: No corrections for clustering or multiple comparisons were needed. The means and standard deviations for SAT-10 scale scores were not reported by the authors. 
However, the authors standardized student test scores using national norms and recalibrated them to have a mean of 0 and a standard deviation of 1 , which means their results 
are in effect size units. Mean difference and effect sizes were regression-adjusted for pretests, grade at baseline, gender, age, receipt of free or reduced-price lunch, race, and 
whether the student applied with a sibling; family-level controls including fixed effects for the center to which the student applied; and a set of baseline self-perceptions of abilities, 
peer academic support, and general adult support. The regression also includes cohort fixed effects (probabilities of random assignment differed by cohort) and standard errors 
clustered by family (the level of randomization). 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from 
the authors or related studies) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s 
assessment of whether the study meets WWC evidence standards and summarizes the study findings following WWC conventions 
for reporting evidence on effectiveness. This study was reviewed using the single study review protocol, version 2.0. 

2 There were 21 outcomes included in the study that are not described in this WWC report. See the table notes in Appendix B 
for more information. 

3 A survey was administered to parents of children in the spring of their fourth year of participation, as students were taking the 
posttest exams. In the survey, parents were asked to name the high schools to which their children had applied, been admitted, 
and at which they matriculated as of the spring of the eighth grade. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, April). 

WWC review of the report: Staying on track: Testing Higher Achievement’s long-term impact on academic 
outcomes and high school choice. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% Ip < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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